Remote support via visualizations of instructional procedures

ABSTRACT

A system may include a processor and a memory including computer-executable code that, when executed by the processor, causes the processor to receive a service request from a first device and to transmit a workflow dataset associated with an industrial automation device to the first device and the second device. The service request may cause a computing system to communicatively couple the first device to a second device. The workflow dataset may include one or more instructions associated with one or more operations for the industrial automation device and one or more virtual objects associated with the one or more instructions and the industrial automation device. The memory may include computer-executable code that, when executed by the processor, causes the processor to receive an input to the image data from the second device and to transmit the one or more additional visualizations to the first device for display.

BACKGROUND

The disclosure relates generally to the design of industrial systems.More particularly, embodiments of the present disclosure are related tosystems and methods for displaying or modifying visualizationsassociated with an industrial automation device or an industrial systembased on detected user input.

Augmented reality (AR) devices provide layers of computer-generatedcontent superimposed (e.g., overlaid) on a visualization of a real-worldenvironment to a user via a display. That is, an AR environment mayprovide a user with a combination of real-world content andcomputer-generated content via the display. Augmented reality devicesmay include, for example, a head mounted device, smart glasses, avirtual retinal display, a contact lens, a computer, or a hand-helddevice, such as a mobile phone or a tablet. As AR devices become morewidely available, these devices may be used to assist operators inindustrial automation environments to perform certain tasks. As such, itis recognized that improved systems and methods for communicating to theoperators how to perform certain operations in the real-world via the ARenvironment may better enable the operators to perform their jobfunctions.

This section is intended to introduce the reader to various aspects ofart that may be related to various aspects of the present techniques,which are described and/or claimed below. This discussion is believed tobe helpful in providing the reader with background information tofacilitate a better understanding of the various aspects of the presentdisclosure. Accordingly, it should be understood that these statementsare to be read in this light, and not as admissions of prior art.

BRIEF DESCRIPTION

A summary of certain embodiments disclosed herein is set forth below. Itshould be understood that these aspects are presented merely to providethe reader with a brief summary of these certain embodiments and thatthese aspects are not intended to limit the scope of this disclosure.Indeed, this disclosure may encompass a variety of aspects that may notbe set forth below.

In one embodiment, a system may include a processor and a memoryincluding computer-executable code that, when executed by the processor,causes the processor to receive a service request from a first device.The service request may cause a computing system to communicativelycouple the first device to a second device. The memory may includecomputer-executable code that, when executed by the processor, causesthe processor to transmit a workflow dataset associated with anindustrial automation device to the first device and the second device.The workflow dataset may include one or more instructions associatedwith one or more operations for the industrial automation device and oneor more virtual objects associated with the one or more instructions andthe industrial automation device. The workflow dataset may cause thefirst device and the second device to display one or more visualizationsrepresentative of the one or more instructions, the one or more virtualobjects, or both on image data depicted on a first display and on asecond display associated with the first device and the second device,respectively. The memory may include computer-executable code that, whenexecuted by the processor, causes the processor to receive an input tothe image data from the second device, where the input may include oneor more additional visualizations, and causes the processor to transmitthe one or more additional visualizations to the first device fordisplay via the first display.

In another embodiment, a method may include receiving, via a processor,a service request from a first device. The service request may cause acomputing system to communicatively couple the first device to a seconddevice. The method may include transmitting, via the processor, aworkflow dataset associated with an industrial automation device to thefirst device and the second device. The workflow dataset may include oneor more instructions associated with one or more operations for theindustrial automation device and one or more virtual objects associatedwith the one or more instructions and the industrial automation device.The workflow dataset may cause the first device and the second device todisplay one or more visualizations representative of the one or moreinstructions or the one or more virtual objects on image data depictedon a first display and on a second display associated with the firstdevice and the second device, respectively. The method may includereceiving, via the processor, a first input to the image data from thesecond device, where the first input may include one or more additionalvisualizations. The method may include transmitting, via the processor,the one or more additional visualizations to the first device fordisplay via the first display.

In yet another embodiment, a tangible, non-transitory computer-readablemedium may store instructions executable by a processor of a computingdevice that, when executed by the processor, cause the computing deviceto perform operations including receiving a service request from a firstdevice, where the service request may cause a computing system tocommunicatively couple the first device to a second device. Theinstructions may, when executed by the processor, cause the computingdevice to perform operations including transmitting a workflow datasetassociated with an industrial automation device to the first device andthe second device. The workflow dataset may include one or moreinstructions associated with one or more operations for the industrialautomation device and one or more virtual objects associated with theone or more instructions and the industrial automation device. Theworkflow dataset may cause the first device and the second device todisplay one or more visualizations representative of the one or moreinstructions or the one or more virtual objects on image data depictedon a first display and on a second display associated with the firstdevice and the second device, respectively. The instructions may, whenexecuted by the processor, cause the computing device to performoperations including receiving a first input to the image data from thesecond device, where the first input may include one or more additionalvisualizations. The instructions may, when executed by the processor,cause the computing device to perform operations including transmittingthe one or more additional visualizations to the first device fordisplay via the first display.

DRAWINGS

These and other features, aspects, and advantages of the presentdisclosure may become better understood when the following detaileddescription is read with reference to the accompanying drawings in whichlike characters represent like parts throughout the drawings, wherein:

FIG. 1 is a perspective view of an example industrial automation system,in accordance with an embodiment;

FIG. 2 is a block diagram of an industrial control system that operatesa portion of the industrial automation system of FIG. 1 , in accordancewith an embodiment;

FIG. 3 is a block diagram of an electronic device receiving an augmentedreality (AR) user experience (UX) workflow from an AR UX system, inaccordance with an embodiment;

FIG. 4 is a flowchart of a method for operating the electronic device ofFIG. 3 to receive and display the AR UX workflow, in accordance with anembodiment;

FIG. 5 is an illustration of a first example graphical user interface(GUI) of the electronic device of FIG. 3 , in accordance with anembodiment;

FIG. 6 is an illustration of a second example GUI of the electronicdevice of FIG. 3 , in accordance with an embodiment;

FIG. 7 is an illustration of a third example GUI of the electronicdevice of FIG. 3 , in accordance with an embodiment;

FIG. 8 is an illustration of an example visualization presented via adisplay of the electronic device of FIG. 3 , in accordance with anembodiment;

FIG. 9 is an illustration of an example visualization of informationpresented via the display of the electronic device of FIG. 3 , inaccordance with an embodiment;

FIG. 10 is an illustration of an example visualization of instructionspresented via the display of the electronic device of FIG. 3 , inaccordance with an embodiment;

FIG. 11 is an illustration of an example visualization of a list oftools presented via the display of the electronic device of FIG. 3 , inaccordance with an embodiment;

FIG. 12 is an illustration of an example visualization of an augmentedreality industrial component presented via the display of the electronicdevice of FIG. 3 using image data representative of real objects, inaccordance with an embodiment;

FIG. 13 is a flowchart of a method for displaying the AR UX workflow viathe electronic device of FIG. 3 , in accordance with an embodiment;

FIG. 14 is an illustration of an example visualization of notespresented via the display of the electronic device of FIG. 3 , inaccordance with an embodiment;

FIG. 15 is an illustration of an example visualization depicting achange in the industrial component presented via the display of theelectronic device of FIG. 3 , in accordance with an embodiment;

FIG. 16 is an illustration of an example visualization of additionalnotes presented via the display of the electronic device of FIG. 3 , inaccordance with an embodiment;

FIG. 17 is an illustration of an example visualization of a first stepof a workflow presented via the display of the electronic device of FIG.3 , in accordance with an embodiment;

FIG. 18 is an illustration of an example visualization of a second stepof a workflow presented via the display of the electronic device of FIG.3 , in accordance with an embodiment;

FIG. 19 is an illustration of an example visualization of a third stepof a workflow presented via the display of the electronic device of FIG.3 , in accordance with an embodiment;

FIG. 20 is an illustration of an example visualization of a fourth stepof a workflow presented via the display of the electronic device of FIG.3 , in accordance with an embodiment;

FIG. 21 is an illustration of an example visualization of a fifth stepof a workflow presented via the display of the electronic device of FIG.3 , in accordance with an embodiment;

FIG. 22 is an illustration of an example visualization of a sixth stepof a workflow presented via the display of the electronic device of FIG.3 , in accordance with an embodiment;

FIG. 23 is a block diagram of the electronic device of FIG. 3 receivinga modified AR UX workflow from an additional electronic deviceassociated with a customer service representative, in accordance with anembodiment; and

FIG. 24 is an illustration of an example visualization that depicts userinput graphics presented via the display of the electronic device ofFIG. 3 , in accordance with an embodiment.

DETAILED DESCRIPTION

One or more specific embodiments of the present disclosure will bedescribed below. In an effort to provide a concise description of theseembodiments, all features of an actual implementation may not bedescribed in the specification. It should be appreciated that in thedevelopment of any such actual implementation, as in any engineering ordesign project, numerous implementation-specific decisions must be madeto achieve the developers' specific goals, such as compliance withsystem-related and business-related constraints, which may vary from oneimplementation to another. Moreover, it should be appreciated that sucha development effort might be complex and time consuming, but wouldnevertheless be a routine undertaking of design, fabrication, andmanufacture for those of ordinary skill having the benefit of thisdisclosure.

When introducing elements of various embodiments of the presentdisclosure, the articles “a,” “an,” “the,” and “said” are intended tomean that there are one or more of the elements. The terms “comprising,”“including,” and “having” are intended to be inclusive and mean thatthere may be additional elements other than the listed elements. One ormore specific embodiments of the present embodiments described hereinwill be described below. In an effort to provide a concise descriptionof these embodiments, all features of an actual implementation may notbe described in the specification. It should be appreciated that in thedevelopment of any such actual implementation, as in any engineering ordesign project, numerous implementation-specific decisions must be madeto achieve the developers' specific goals, such as compliance withsystem-related and business-related constraints, which may vary from oneimplementation to another. Moreover, it should be appreciated that sucha development effort might be complex and time consuming, but wouldnevertheless be a routine undertaking of design, fabrication, andmanufacture for those of ordinary skill having the benefit of thisdisclosure.

The present disclosure is generally directed towards an interactiveaugmented reality (AR) user experience (UX) system that may display oneor more visualizations of a combination of real-world andcomputer-generated content in an AR environment to an operator.Operators of industrial automation systems may sometimes performoperations to industrial automation devices of the industrial automationsystem, such as maintenance activities and/or replacement activities.The visualizations may be of a portion of an industrial automationsystem and may help the operator when performing an operation to theportion of the industrial automation system. These operations may beperformed in accordance with device-specific procedures (e.g., sequencesof operations). For example, a first motor drive may be powered offusing a procedure that is different from the procedure for a secondmotor drive due to the different components connected to each motordrive. The operator may be expected to memorize each of thedevice-specific procedures and/or reference hard copies of procedureswhile performing the operations. Even if the operator does not intermixprocedures and devices, the operator may have a difficult timeidentifying individual portions of the industrial automation device thathave changed or have been adjusted during the procedure. Thus, it may bedesirable to provide an AR UX system for the operator to reference whileperforming operations to the industrial automation devices. The AR UXsystem may reduce a likelihood of operator error by presentingdevice-specific workflows to the operator when the operator is preparingto adjust operator of an industrial automation device. The AR UX systemmay also improve operations of the industrial automation system byreducing downtime of the industrial automation system for maintenanceactivities, for replacement activities and, and/or other suitableactivities by improving operator confidence in performing operations onthe industrial automation device.

Keeping this in mind, the AR UX system may generate one or morevisualizations for arrangement and presentation as AR UX workflows. AnAR UX workflow may include steps associated with performing variousmaintenance procedures or industrial operation activities by an operatorvia a computing device associated with the operator. That is, thecomputing device (e.g., mobile phone) may present a live image datastream (e.g., images captured in real-time from an image sensor of thedevice, video feed) of an industrial automation device and overlayinstructions or information related to the industrial automation deviceon the image data stream. In some cases, the AR UX system may presentone or more virtual objects that are overlaid into the image datastream. The virtual objects may be manipulated or animated as part ofthe AR UX workflow to show portions (e.g., steps, sequential operations)associated with the AR UX workflow. For example, the interactive AR UXsystem may guide an operator through a maintenance procedure bydisplaying or modifying one or more virtual objects in the visualizationassociated with the AR environment presented to the user. That is, theinteractive AR UX system may depict a visualization indicative of a nextcomponent to remove a visualization representative of a next procedureoperation being performed on the virtual object. For example, the AR UXsystem may render an animated visualization depicting a screw beingremoved from a virtual object that represents an enclosure to illustratehow a screen from an actual motor drive is removed. In some embodiments,modifying a visualization or modifying a virtual object in avisualization may include generating an additional visualization todepict the relevant instruction or action and displaying the additionalvisualization.

Although the AR UX workflow is described herein as includingcomputer-generated content to be displayed via a visual modality on adisplay of the computing device, it should be noted that the AR UXworkflow may provide computer-generated content via other types ofsensory modalities. For example, the computer-generated content may bepresented to a user via an auditory modality, a haptic modality, asomatosensory modality, an olfactory modality, or the like. Furthermore,the AR UX system may present instructions or may guide an operator whileperforming an operation through a combination of modalities, such as acombination of audible cues and visual cues, a combination ofvisualizations and haptic feedback, or the like. For example, when theAR UX system causes the operator device to render an image that isoutside a current frame shown via the live image data stream, such as ona backside of an industrial device when the front of the industrialdevice is visible in the display of the operator device current frame,the AR UX system may provide directional haptic feedback to guide theoperator to move the operator device to the area of the real-worldcorresponding to a location of the image being rendered (e.g., thebackside of the industrial device).

In an example embodiment, the AR UX system may guide the operator usinga combination of text and image cues presented on the computing deviceof the operator. In this way, while performing a maintenance activity onan example industrial automation device, such as powering off acomponent, the operator may receive step-by-step instructions accordingto the AR UX workflow regarding how to perform the maintenance activityon the industrial automation device (e.g., for the particular model andversion of the drive). The AR UX system may present a visualization ofthe industrial automation device (e.g., virtual object) and manipulatethe visualization of the industrial automation device to guide theoperator through performance of the maintenance activity. For example,the AR UX system may sequentially show the operator a first location tocheck a status, a second location to disconnect a first switch, a thirdlocation to disconnect a second switch, and a fourth location to verifyan electrical status of the drive used to verify the drive is poweredoff. In some cases, the AR UX system may identify a time that anoperation for a current instruction was completed by the operator and,in response to the identification, the AR UX system may automaticallyadvance to a subsequent instruction.

To access the AR UX workflow for a particular industrial automationdevice, the computing device of the operator may request access from anAR UX system that has access to an AR UX workflow database. Thecomputing device may receive one or more AR UX workflows from the AR UXsystem if more than one AR UX workflows results from a query used by theAR UX system to search the AR UX workflow database. The computing devicemay transmit the selection to the AR UX system and receive the AR UXworkflow corresponding to the selection for presentation to the operatorvia a display of the computing device.

Furthermore, in some embodiments, the computing device may receive oneor more inputs from the operator specifying parameters for a query ofthe AR UX workflow database. For example, the inputs may specifyparameters associated with a model type of the industrial automationdevice, a type of device associated with the industrial automationdevice, an environment that the industrial automation device is operatedwithin, an operation type defining a particular kind of maintenanceactivity to be performed to the industrial automation device (e.g.,replace component of industrial automation device, power off industrialautomation device), or the like. The parameters may be applied to thequery of the workflow database to narrow results of the query beforepresenting the results via the display to the operator.

Additionally or alternatively, the computing device of the operator maybe associated with a profile. The profile may include parameters thatcorrespond to or define an identity of the operator, an identity of theindustrial automation system (e.g., a company that owns or operates theindustrial automation system), or the like. The profile may be used bythe AR UX system to pre-filter a query of the workflow database, suchthat a subset of workflows relevant to the industrial automation systemand/or operator are searched by the AR UX system against the parameters.In some embodiments, the operator may input the parameters of the queryinto the computing device using drop-down menus, radio buttons, or thelike, from a defined set of options. Thus, in some cases, the AR UXsystem may instruct the device to present a subset of options to theoperator based at least in part on the profile corresponding to theoperator. In this way, the operator is unable to select options that theprofile of the operator is ineligible to select. The subset of optionsfor parameters presented to the operator may be based on the profile.The profile may define permissions of the operator, such as operationsor equipment that the operator is eligible to adjust. The profile mayalso include an indication of portions of the industrial automationsystem that the operator works within or devices that are includedwithin the industrial automation system to filter irrelevant industrialautomation devices out from the query. Profile filtering of the workflowdatabase may improve a speed of the query used by the AR UX system whenquerying the workflow database based on parameters by reducing a subsetof workflows that the AR UX system searches with the query. Moreover,profile filtering and other features described herein may reduce theamount of processing power employed by the AR UX system, therebyimproving the operation of the AR UX system.

After the AR UX system transmits the AR UX workflow to the computingdevice for presentation to the operator, the computing device mayautomatically update visualizations presented via the display of thecomputing device to sequentially progress through the instructions ofthe procedure corresponding to the AR UX workflow. In some embodiments,the AR UX workflow may include messages or preliminary information. Thecomputing device may present the messages and/or the preliminaryinformation via the display before presenting the instructions via thedisplay.

Furthermore, in some embodiments, the AR UX workflow may besimultaneously presented to two computing devices. For example, theoperator may request customer service support associated with anoperation to an industrial automation device. In response to therequest, the AR UX system may provide a computing device of the operatorand a computing device of a customer service representative apresentation of the same AR UX workflow. The same AR UX workflow beingpresented to both the operator and the customer service representativemay improve an ability of the customer service representative to guidethe operator through the procedure corresponding to the AR UX workflow.Furthermore, in some embodiments, the computing device of the customerservice representative may receive an input to instruct generation ofvisualizations (e.g., manually entered visualizations) to be overlaid onthe AR UX workflows. The overlaid visualizations may be rendered on thevisualizations of the AR UX workflow, such that the computing device ofthe operator and the computing device of the customer servicerepresentative render the overlaid visualizations at a substantiallysimilar time. In this way, the customer service representative may guidethe operator through performing the procedure using visual indicatorsdrawn in real-time on various portions of the AR UX workflow. Additionaldetails regarding the interactive AR UX workflow and the AR UX systemare described in more detail below with reference to FIGS. 1-24 . It isnoted, that as referred to herein, the AR UX workflow may be associatedwith an AR UX workflow dataset that includes information that enables acomputing device to sequentially render AR UX visualizations that conveyone or more AR UX workflow instructions.

By way of introduction, FIG. 1 is a perspective view of an exampleindustrial automation system 10 employed by a food manufacturer. Itshould be noted that although the example industrial automation system10 of FIG. 1 is directed at a food manufacturer, the present embodimentsdescribed herein may be employed within any suitable industry, such asautomotive, mining, hydrocarbon production, manufacturing, and the like.The following brief description of the example industrial automationsystem 10 employed by the food manufacturer is provided herein to helpfacilitate a more comprehensive understanding of how the embodimentsdescribed herein may be applied to industrial devices to significantlyimprove the operations of the respective industrial automation system.As such, the embodiments described herein should not be limited to beapplied to the example depicted in FIG. 1 .

Referring now to FIG. 1 , the example industrial automation system 10for a food manufacturer may include silos 12 and tanks 14. The silos 12and the tanks 14 may store different types of raw material, such asgrains, salt, yeast, sweeteners, flavoring agents, coloring agents,vitamins, minerals, and preservatives. In some embodiments, sensors 16may be positioned within or around the silos 12, the tanks 14, or othersuitable locations within the industrial automation system 10 to measurecertain properties, such as temperature, mass, volume, pressure,humidity, and the like.

The raw materials may be provided to a mixer 18, which may mix the rawmaterials together according to a specified ratio. The mixer 18 andother machines in the industrial automation system 10 may employ certainindustrial automation devices 20 to control the operations of the mixer18 and other machines. The industrial automation devices 20 may includecontrollers, input/output (I/O) modules, motor control centers, motors,human machine interfaces (HMIs), operator interfaces, contactors,starters, sensors 16, actuators, conveyors, drives, relays, protectiondevices, switchgear, compressors, sensor, actuator, firewall, networkswitches (e.g., Ethernet switches, modular-managed, fixed-managed,service-router, industrial, unmanaged, etc.) and the like.

The mixer 18 may provide a mixed compound to a depositor 22, which maydeposit a certain amount of the mixed compound onto conveyor 24. Thedepositor 22 may deposit the mixed compound on the conveyor 24 accordingto a shape and amount that may be specified to a control system for thedepositor 22. The conveyor 24 may be any suitable conveyor system thattransports items to various types of machinery across the industrialautomation system 10. For example, the conveyor 24 may transportdeposited material from the depositor 22 to an oven 26, which may bakethe deposited material. The baked material may be transported to acooling tunnel 28 to cool the baked material, such that the cooledmaterial may be transported to a tray loader 30 via the conveyor 24. Thetray loader 30 may include machinery that receives a certain amount ofthe cooled material for packaging. By way of example, the tray loader 30may receive 25 ounces of the cooled material, which may correspond to anamount of cereal provided in a cereal box.

A tray wrapper 32 may receive a collected amount of cooled material fromthe tray loader 30 into a bag, which may be sealed. The tray wrapper 32may receive the collected amount of cooled material in a bag and sealthe bag using appropriate machinery. The conveyor 24 may transport thebagged material to case packer 34, which may package the bagged materialinto a box. The boxes may be transported to a palletizer 36, which maystack a certain number of boxes on a pallet that may be lifted using aforklift or the like. The stacked boxes may then be transported to ashrink wrapper 38, which may wrap the stacked boxes with shrink-wrap tokeep the stacked boxes together while on the pallet. The shrink-wrappedboxes may then be transported to storage or the like via a forklift orother suitable transport vehicle.

To perform the operations of each of the devices in the exampleindustrial automation system 10, the industrial automation devices 20may be used to provide power to the machinery used to perform certaintasks, provide protection to the machinery from electrical surges,prevent injuries from occurring with human operators in the industrialautomation system 10, monitor the operations of the respective device,communicate data regarding the respective device to a supervisorycontrol system 40, and the like. In some embodiments, each industrialautomation device 20 or a group of industrial automation devices 20 maybe controlled using a local control system 42. The local control system42 may include receive data regarding the operation of the respectiveindustrial automation device 20, other industrial automation devices 20,user inputs, and other suitable inputs to control the operations of therespective industrial automation devices 20.

FIG. 2 is a block diagram of an operator device 54 that may be employedin any suitable industrial automation system 10 to access or interfacewith other industrial automation device 20 and/or computing devices. Forexample, the operator device 54 may include a communication component56, a processor 58, a memory 60, a storage 62, input/output (I/O) ports64, an image sensor 66 (e.g., a camera), a location sensor 68, a display70, additional sensors (e.g., vibration sensors, temperature sensors),and the like. The communication component 56 may be a wireless or wiredcommunication component that may facilitate communication between theindustrial automation device 20, cloud-based computing systems, andother communication capable devices.

The processor 58 may be any type of computer processor or microprocessorcapable of executing computer-executable code. The processor 58 may alsoinclude multiple processors that may perform the operations describedbelow. The memory 60 and the storage 62 may be any suitable articles ofmanufacture that can serve as media to store processor-executable code,data, or the like. These articles of manufacture may representcomputer-readable media (e.g., any suitable form of memory or storage)that may store the processor-executable code used by the processor 58 toperform the presently disclosed techniques. Generally, the processor 58may execute software applications that include programs that enable auser to track and/or monitor operations of the industrial automationdevice 20 via a local or remote communication link. That is, thesoftware applications may communicate with the operator device 54 andgather information associated with the industrial automation device 20as determined by the operator device 54, via the sensors 16 disposed onthe industrial automation device 20 and the like.

The memory 60 and the storage 62 may also be used to store the data,analysis of the data, the software applications, and the like. Thememory 60 and the storage 62 may represent non-transitorycomputer-readable media (e.g., any suitable form of memory or storage)that may store the processor-executable code used by the processor 58 toperform various techniques described herein. It should be noted thatnon-transitory merely indicates that the media is tangible and not asignal.

In one embodiment, the memory 60 and/or storage 62 may include asoftware application that may be executed by the processor 58 and may beused to monitor, control, access, or view one of the industrialautomation devices 20. The software application may enable the operatordevice 54 to perform various functionalities, such as track statisticsof the industrial automation device 20, access reasons for placing theindustrial automation device 20 offline stored by the local controlsystem 42, and so forth.

The I/O ports 64 may be interfaces that couple to other peripheralcomponents such as input devices (e.g., keyboard, mouse), sensors,input/output (I/O) modules, and the like. I/O modules may enable thecomputing device or other operator devices 54 to communicate with theindustrial automation device 20 or other devices in the industrialautomation system 10 via the I/O modules.

The image sensor 66 may include any image acquisition circuitry such asa digital camera capable of acquiring digital images, digital videos, orthe like. The location sensor 68 may include circuitry designed todetermine a physical location of the operator device 54. In oneembodiment, the location sensor 68 may include a global positioningsystem (GPS) sensor that acquires GPS coordinates for the operatordevice 54.

The display 70 may depict visualizations associated with software orexecutable code being processed by the processor 58. In one embodiment,the display 70 may be a touch display capable of receiving inputs (e.g.,parameter data for operating the industrial automation device 20) from auser of the operator device 54. As such, the display 70 may serve as auser interface to communicate with the industrial automation device 20.The display 70 may be used to display a graphical user interface (GUI)for operating the industrial automation device 20, for tracking themaintenance of the industrial automation device 20, and the like. Thedisplay 70 may be any suitable type of display, such as a liquid crystaldisplay (LCD), plasma display, or an organic light emitting diode (OLED)display, for example. Additionally, in one embodiment, the display 70may be provided in conjunction with a touch-sensitive mechanism (e.g., atouch screen) that may function as part of a control interface for theindustrial automation device 20 or for a number of pieces of industrialautomation equipment in the industrial automation system 10 to controlthe general operations of the industrial automation system 10. In someembodiments, the user interface may be characterized as a HMI, ahuman-interface machine, or the like.

Although the components described above have been discussed with regardto the operator device 54, it should be noted that similar componentsmay make up other computing devices described herein, such as the localcontrol system 42. Further, it should be noted that the listedcomponents are provided as example components and the embodimentsdescribed herein are not to be limited to the components described withreference to FIG. 2 . For example, other computing or control devices ofthe industrial automation system 10 may include one or more componentsthat the operator device 54 includes.

To elaborate further on the operator device 54, FIG. 3 is a blockdiagram of the operator device 54 receiving an augmented reality (AR)user experience (UX) workflow dataset from an AR UX system 82, where theAR UX workflow dataset may include data corresponding to virtual objects(e.g., virtual products) as part of a series or sequence of sequentialvisualizations (e.g., AR UX workflow visualizations) to communicatesequentially presented instructions (e.g., AR UX workflow instructions)overlaid on real-time obtained and displayed images (e.g., image datafeed) to communicate an operation or procedure (e.g., experience) to theoperator 50 via dynamically generated instructions. Each AR UX workflowinstruction of an AR UX workflow dataset may correspond to an AR UXvisualization that may include one or more virtual objects, where one ormore of the one or more virtual objects may be animated to convey orshow steps for completing the AR UX workflow instruction. The AR UXworkflow may be a 360-degree presentation to illustrate steps of aprocedure to an operator 88, such that the AR UX workflow is presentedvia manipulation and/or emphasis of a virtual product depicted in areal-world location via an AR environment using image datarepresentative of visualizations of industrial automation devicesoverlaid on image data representative of the real-world environmentviewable by the operator device 54.

It is noted that the AR UX system 82 may be a computing device, hostedon a computing device, hosted on a cloud-computing device, or the like,and thus may include one or more components described with reference tothe operator device 54 in FIG. 2 . For example, the AR UX system 82 mayinclude a processor 58 for performing operations and executinginstructions stored in a memory 60. In this way, the processor 58 of theAR UX system 82 may retrieve an AR UX workflow dataset and initiatetransmission of the AR UX workflow dataset to the operator device 54 viacommunication component 56 and/or I/O ports 64.

The operator device 54 may use the AR UX workflow dataset to render avisualization 84 that includes a virtual representation of an industrialautomation device 86 (e.g., virtual industrial automation device,virtual product) in an augmented reality (AR) environment as an AR UXworkflow visualization. In the illustrated embodiment, the ARenvironment may refer to the visualization 84 having a combination ofcomputer-generated and real-world content displayed to the operator 50via the operator device 54. Although a handheld operator device 54 isemployed within the illustrated embodiment, it should be noted that, inother embodiments, other suitable types of displays or devices may beemployed. For example, the operator device 54 may take the form of smartglasses, a virtual retinal display, one or more contact lenses, acomputer, a mobile device, or any other suitable electronic displaydevice for presenting AR UX workflow visualizations to the operator 50.In any case, the operator device 54 may display the visualization 84that includes a virtual industrial automation device 86 to the operator50 via the operator device 54. The visualization 84 may be superimposedcomputer-generated content (e.g., images or sounds) over real-worldcontent (e.g., video, images or sounds) of the user's environment inreal-time (e.g., used to present AR UX workflow visualizationsindicative of AR UX workflow instructions).

In the illustrated embodiment, the operator device 54 may display avisualization 84 that includes a virtual representation of a motor drive86. However, it should be noted that the illustrated embodiment isintended to be non-limiting and that the operator device 54 renderingthe AR UX workflow dataset may display a visualization 84 that includesother virtual industrial automation devices, or parts thereof, that maybe employed within an industrial system. For example, the industrialautomation devices 20 may include controllers, input/output (I/O)modules, motor control centers, motors, valves, actuators, temperatureelements, pressure sensors, HMIs, operator interfaces, contactors,starters, sensors, drives, relays, protection devices, switchgear,compressors, network switches (e.g., Ethernet switches, modular-managed,fixed-managed, service-router, industrial, unmanaged), data centers,conveyor sections, movers, and the like, and thus the AR UX workflowdataset may be used to present a procedure associated with any one ofthe above-listed devices or other suitable industrial devices.

The visualization 84 presented by the operator device 54 may be aportion of the AR UX workflow dataset. The operator device 54 mayrequest the AR UX workflow dataset from the AR UX system 82 and obtainimage data from the AR UX workflow dataset to present variousvisualizations 84 to the operator 50 via the operator device 54.

To obtain the image data, the operator device 54 may communicate withthe AR UX system 82 through a network 90 and retrieve the image data ofthe AR UX workflow dataset based on input data. The network 90 mayinclude any wired or wireless network that may be implemented as a localarea network (LAN), a wide area network (WAN), and the like. It shouldbe noted that any suitable network may be employed in the embodimentsdescribed herein. Indeed, other industrial communication networkprotocol, such as ETHERNET/IP®, CONTROLNET®, DEVICENET®, and the like,may also be used. In any case, the network 90 may permit the exchange ofdata in accordance with a protocol. For example, the network 90 maypermit the transmission of the AR UX workflow dataset from the AR UXsystem 82 to the operator device 54.

Before a visualization representative of a AR UX workflow instructionmay be presented via the operator device 54, a particular AR UX workflowdataset may be selected by the operator device 54 (e.g., by input fromthe operator 50). Selecting the AR UX workflow dataset may notify the ARUX system 82 to initiate a download of data corresponding to theselected AR UX workflow dataset to the operator device 54. To helpselect the AR UX workflow dataset, the AR UX system 82 may transmit anindication of available AR UX workflow datasets to the operator device54, such that a selection of an AR UX workflow dataset may be made basedon the available AR UX workflows. In this way, the AR UX system 82 mayreceive a request for an AR UX workflow from the operator device 54,determine available AR UX workflow datasets for presentation on theoperator device 54, generate indications of the available AR UXworkflows (e.g., a list of AR UX workflow titles), transmit theindications of the available AR UX workflow datasets to the operatordevice 54, receive a selection of one of the available AR UX workflowdatasets via the operator device 54 selecting one of the indications,and transmitting at least a portion of data corresponding to theselected available AR UX workflow dataset to the operator device 54.Transmitting the selected available AR UX workflow dataset after theoperator device 54 is presented with options for available AR UXworkflow datasets may improve operation of the AR UX system 82 and/orthe operator device 54 by reducing an amount of data transmitted betweenthe AR UX system 82 and the operator device 54 when determining theselected available AR UX workflow.

The operator 50 may enter query parameters into the operator device 54to initiate a search of AR UX workflow datasets. The AR UX system 82 mayreceive the query parameters from the operator device 54 and use thequery parameters to search for available AR UX workflows relevant to thequery parameters. Query parameters may specify a type of equipment, atype of operator, a unit of the industrial automation system 10, or thelike to further specify the operation that the operator 50 desires in anAR UX workflow to use to help perform a respective task. The AR UXsystem 82 may use the query parameters to filter through AR UX workflowdatasets stored in an AR UX workflow database 92. AR UX workflowdatasets determined relevant based on the query parameters may betransmitted to the operator device 54 as the available AR UX workflowdatasets from which the operator selects.

The AR UX system 82 may also perform additional filtering beforequerying the AR UX workflow database 92. The additional filtering mayreduce a number of workflows to be searched via the query. The AR UXsystem 82 may reference stored profiles to determine a subset of AR UXworkflow datasets that are relevant to be queried. For example, aprofile may include information about a permission or skill level of theoperator 50, such that if the operator 50 is not trained on anindustrial automation device 20 or is otherwise unpermitted to adjustoperation of the industrial automation device 20, the AR UX system 82may not query AR UX workflow datasets related to the industrialautomation device 20 since those AR UX workflow datasets may beirrelevant to the operator 50. The AR UX system 82 may also filter AR UXworkflow datasets based on profiles for the industrial automation system10. In this way, a profile for the industrial automation system 10 maystore indications of related equipment (e.g., part of an overallsystem), model numbers of the related equipment, operational preferencesfor the related equipment, or the like for the industrial automationsystem 10. Thus, when any operator device 54 of the industrialautomation system 10 requests an AR UX workflow, the AR UX system 82 mayquery AR UX workflow datasets associated with equipment related oroperations performed by the industrial automation system 10 withoutquerying AR UX workflow datasets associated with equipment unrelated oroperations not performed by the industrial automation system 10.Profiles may be stored in a profile database 94 accessible by the AR UXsystem 82.

The AR UX system 82 may also filter the AR UX workflow datasets based atleast in part on location anchors. A location anchor may be a device(e.g., industrial automation device 20) disposed at a known locationwith the industrial automation system 10. Using location information toperform filtering may enable the AR UX system 82 to query a subset of ARUX workflow datasets that are associated with industrial automationdevices 20 that are located closer to or within a threshold distancefrom the operator device 54.

Location anchors may be used to determine a location of the operatordevice 54 based on the location of the operator device 54 relative toone or more location anchors. For example, time-of-flight calculationsmay be used to associate distances between a location anchor and theoperator device 54 to a duration of time used to transmit a locatingsignal between the operator device 54 and a location anchor. In thisway, the operator device 54 may transmit a first signal to a locationanchor at a first time and receive a second signal from the locationanchor at a second time. The operator device 54 may determine a durationof time between the first time and the second time and may correlate theduration of time to the location of the operator device 54. In someembodiments, the location sensor 68 may use Global Positioning System(GPS) information to locate the operator device 54 within the industrialautomation system 10.

When a location of the operator device 54 is known, the AR UX system 82may not query portions of the AR UX workflow database 92 that correspondto devices disposed a threshold distance away from the operator device54. Industrial automation devices 20 disposed a distance greater thanthe threshold distance away from the operator device 54 may be unlikelyto be selected by the operator 50 and/or may be considered irrelevantfor purposes of querying AR UX workflow datasets. Location anchors maybe stored in a location anchor database 96. Each of the databases 92,94, 96 may be part of a AR UX system database 98 accessed by the AR UXsystem 82 when performing operations to provide the operator device 54with an AR UX workflow dataset.

The operator device 54 may determine its own location (and transmit itslocation to the AR UX system 82) and/or the AR UX system 82 maydetermine the location of the operator device 54. After the AR UX system82 has the location of the operator device 54, the AR UX system 82 maysearch for industrial automation devices 20 within a same region as theoperator device 54 or within a threshold number of regions adjacent to aregion that the operator device 54 is disposed. The AR UX system 82 mayquery the AR UX system database 98 to find AR UX workflow datasets thatcorrespond to the industrial automation devices 20 determined to bewithin regions suitably near to the operator device 54 (e.g., within thesame region, within a threshold number of regions from the operatordevice 54). The AR UX system 82 reducing a set of AR UX workflowsdatasets to be queried before querying the AR UX workflow database 92may improve an efficiency of the query and make the querying operationtake relatively less time than without filtering of the AR UX workflowdatasets.

In addition to improving the query of the AR UX system database 98, thelocation of the operator device 54 may be used by the AR UX system 82 toidentify industrial automation devices 20 and/or other features in animage captured by the image sensor 66 of the operator device 54. In thisway, the AR UX system 82 may use the captured image data received fromthe operator device 54 to identify devices (e.g., industrial automationdevices 20, other suitable identifiable devices within an industrialautomation system 10) in the captured image data. When the AR UX system82 identifies the devices in the captured image data, the AR UX system82 may send information associated with the identified devices to theoperator device 54. For example, the operator 50 may desire to view ARUX workflow instructions for an industrial automation device 20 and mayinitiate the query by capturing a photograph of the industrialautomation device 20. The operator device 54 may transmit the image datacorresponding to the photograph to the AR UX system 82. The AR UX system82 may receive the image data and use image processing operations toidentify and match the image data of the industrial automation device 20to known industrial automation devices 20 of the particular industrialautomation system 10 (e.g., identified in the profiles stored in theprofile database 94). Once a match is determined, the AR UX system 82may transmit information corresponding to the now-identified industrialautomation device 20 to the operator device 54 for reference by theoperator 50. For example, in response to matching the image data of theindustrial automation device 20 to data stored in the AR UX systemdatabase 98, the AR UX system 82 may transmit information to theoperator device 54, such as AR UX workflow dataset corresponding to theindustrial automation device 20. In this way, the AR UX system 82 and/orthe operator device 54 may receive image data from the image sensor 66of a likeness of the industrial automation device 20, may compare theimage data to stored image data (e.g., image data stored in AR UXdatabase 98) corresponding to industrial automation products of theindustrial automation system 10, and may determine the industrialautomation device 20 to be one of the industrial automation productsbased at least in part on the image data matching the stored image data(e.g., matching based on an amount of matching or substantially similarimage data being greater than a threshold amount of matching data).

In some cases, the AR UX system 82 may receive an indication of the ARUX workflow dataset to transmit to the operator device 54 from theoperator device 54 as opposed to query parameters. The operator device54, for example, may scan a link, a barcode (e.g., a QR code, a matrixbarcode), or a radio frequency identification tag (RFID tag), or thelike, presented on the industrial automation device 20 to retrieve theAR UX workflow dataset corresponding to the industrial automation device20. For example, the links, barcodes, or the like may include a codeacquired via a scanner, a barcode acquired via the scanner, a digitallink to a website. Furthermore, in some cases, the AR UX system 82 mayaccess maintenance schedules or the like for the industrial automationdevice, and may automatically retrieve the corresponding AR UX workflowdataset to an upcoming or overdue maintenance procedure for transmissionto the operator device 54. The above-described features and more arediscussed below with reference to interactions between the operatordevice 54 and the AR UX system 82, such as to select and provide dataassociated with an AR UX workflow dataset.

To elaborate on AR UX workflow selection operations, FIG. 4 is aflowchart of a method 110 performed by the operator device 54 to receiveand display an AR UX workflow visualization. Although the method 110 isdescribed below as performed by the operator device 54, it should benoted that the method 110 may be performed by any suitable processorthat presents AR UX workflow visualizations. Moreover, although thefollowing description of the method 110 is described in a particularorder, it should be noted that the method 110 may be performed in anysuitable order.

At block 112, the operator device 54 may receive a request for augmentedreality (AR) support. The request may be generated in response to anapplication running or being opened on the operator device 54. In somecases, the request may be generated in response to receiving ordetecting an audio command, such as a voice command from the operator50. When the operator device 54 receives the request for AR support, theoperator device 54 may launch a graphical user interface (GUI) thatenables the operator 50 to enter query parameters that the AR UX systemuses to retrieve AR UX workflow datasets from the AR UX system database98. Query parameters may include inputs that identify a product, aproduct type (e.g., product line), a desired operation to be illustratedvia the AR UX workflow visualizations, or the like.

In this way, at block 114, the operator device 54 may receive an inputidentifying a product, a product type (e.g., product line), a desiredoperation to be illustrated via the AR UX workflow (e.g., a desiredexperience), or the like. Thus, the input may include information tohelp narrow presented AR UX workflow datasets from which the operator 50selects. The product may be an industrial automation device 20 to beoperated on by the operator 50. Examples of product types may include acategory designation, such as motor drive, controller, valve,positioner, or the like. Examples of the products may include specificexamples of motor drives, such as a designated frame or model number, orspecific models of the product types. Examples of the desiredexperiences may include preventive maintenance operations,lock-out/tag-out operations, start-up operations, part replacementoperations, or the like. Desired experiences may specify the type ofoperation that the operator is to perform to the specified productand/or product type. For example, an experience may correspond to amaintenance procedure that includes instructions (e.g., verbal cues,written cues) that, when presented sequentially, increase a likelihoodthat equipment operated on will suitably work or operate as expected(e.g., as in accordance with an original operational intent).Experiences may correspond to procedures used to power-off or shutdownequipment (e.g., industrial automation devices 20) within an industrialautomation system 10, such as to electrically disconnect the equipmentfrom a power supply. Experiences may correspond to procedures used topower-on equipment (e.g., industrial automation devices 20) within anindustrial automation system 10, such as to electrically connect theequipment to a power supply. Some experiences may be performed as partof preventive maintenance operations (e.g., operations performed beforeor in anticipation of equipment malfunction or undesired operation), aspart of responsive maintenance operations (e.g., operations performed inresponse to an equipment malfunction or undesired operation), or thelike. Generally, an AR UX workflow may involve presenting AR UX workflowvisualizations based on AR UX workflow data (e.g., datasets) to conveyor communicate AR UX workflow instructions to an operator to guide theoperator through performance of a product-specific experience (e.g.,operation performed to equipment of an industrial automation system 10,a device-specific experience, an industrial automation device-specificexperience). In this way, since the AR UX workflow visualizationsprovide product-specific AR UX workflow instructions, inputs identifyingthe product, product type, desired experience, or the like may be usedto query and identify a relevant subset (e.g., one or more) of the AR UXworkflow datasets to be transmitted to the operator device 54 for use inpresenting the AR UX workflow visualizations to the operator 50.

In some embodiments, the operator device 54 and/or the AR UX system 82may receive inputs identifying a product based on a location of theoperator device 54, such as a location determined via the locationsensor 68, a location determined using location anchors, or the like.The operator device 54 and/or the AR UX system 82 may also receive aninput identifying the product based at least in part on the operatordevice 54 scanning a code, a barcode, following a link, or the like tothe identifying information. In some embodiments, the industrialautomation device 20 may transmit its identifying information to theoperator device 54, such as in response to the operator device 54 beingwithin a defined proximity to the industrial automation device 20 and/orin response to the operator device 54 requesting the identifyinginformation from the industrial automation device 20. For example, insome embodiments, the industrial automation device 20 may determine atime at which a particular maintenance activity is to be performed onitself and may push a notification to the operator device 54 and/or awearable computing device of the operator 88 (e.g., smart watch) tonotify the operator 88 of the due maintenance activity.

The operator device 54 and/or the AR UX system 82 may additionally oralternatively use detected habits and/or predictive models to determinewhich AR UX workflow is to be requested. For example, the AR UX system82 may determine and reference a stored maintenance pattern for aparticular industrial automation device 20. The stored maintenancepattern may define a regular repetition or periodicity to be used toidentify a time at which a maintenance operation is to be performed. Forexample, the stored maintenance pattern may define that a calibrationoperation to an industrial automation device is to be performed everythirty days. A variety of suitable maintenance operations and/ordurations of time may be stored as a stored maintenance pattern. Usingthe stored maintenance pattern, the AR UX system 82 may determine thatthe operator device 54 is expected to request data associated with theAR UX workflow corresponding to the particular industrial automationdevice 20 at the current time since the current time corresponds to anext operation time of the stored maintenance pattern. Furthermore, insome embodiments, the operator device 54 and/or the AR UX system 82 mayreceive the input identifying the product based at least in part onimage recognition processes that analyze image data of the operatordevice 54 to detect which product is of interest.

At block 116, the operator device 54 may use the input identifying theproduct to generate and send a query to the AR UX system 82. The AR UXsystem 82 may receive the query and use information of the query (e.g.,input parameters of product type, product, desired experience) to searchthe AR UX system database 98. The query may result in the AR UX system82 identifying a subset of AR UX workflow datasets that are relevant tothe input received by the operator device 54. AR UX workflows may beassociated with data, including image data, used by the operator device54 to present visualizations of steps related to a procedure to performvarious types of tasks. Visualizations associated with a respective ARUX workflow dataset may be rendered via a display of the operator device54 to generate a graphical user interface and to provide overlaid imagesthat communicate safety information, tools, lock-out/tag-out (e.g.,electrical isolation) information, procedure steps, or the like to theoperator 50 preparing to operate or actively operating on the industrialautomation device 20.

In some embodiments, the AR UX system 82 may query the AR UX systemdatabase 98 based on a profile from the profile database 94 and/or basedon a location of the operator device 54 in addition to querying the ARUX system database 98 based on the input received by the operator device54. Querying based on the profile may provide a filtered subset of AR UXworkflow datasets even more tailored to the operator device 54, sincethe additional query may enable omission of AR UX workflow datasets thatare irrelevant to the operator 50 based on the permission level of theoperator 50, a subscription level of the industrial automation system10, equipment of the industrial automation system 10, or the like.Furthermore, filtering based on the location of the operator device 54may help provide a filtered subset of AR UX workflow datasets tailoredto a location of the operator device 54, such that AR UX workflowdatasets related to equipment at some distance from the operator device54 (e.g., a threshold distance from the location of the operator device54) are excluded from the subset of AR UX workflow datasets resultingfrom the query. When the subset of the AR UX workflow datasets (or thefiltered subset of the AR UX workflow datasets) is prepared by the AR UXsystem 82, the AR UX system 82 may transmit data associated with thesubset of the AR UX workflow datasets to the operator device 54.

At block 118, the operator device 54 may receive the AR UX workflowdatasets from the AR UX system 82. In some cases, the query results inone AR UX workflow dataset, thus the operator device 54 may receive datafor one AR UX workflow dataset. However, in some cases, the queryresults in multiple AR UX workflow dataset options. In these cases, theoperator device 54 may receive indications of the multiple AR UXworkflow dataset options and provide a selection from the multipleoptions to the AR UX system 82 before downloading the data for theselected AR UX workflow dataset. It is noted that in some cases, datafor multiple AR UX workflow datasets is transmitted to the operatordevice 54 and any unselected workflows from the multiple AR UX workflowdatasets are discarded by the operator device 54.

At block 120, the operator device 54 may display an AR UX workflowvisualization 84 via the display 70. The operator device 54 may useimage data of the AR UX workflow dataset to generate overlaidinstructions and/or visualizations 84 associated with the selectedexperience type. For example, the operator device 54 may generate avisualization 84 of a motor drive according to the AR UX workflowdataset. The AR UX workflow dataset may also define additionaloperations to perform to the motor drive to execute the selectedexperience, such as individual instructions to power off the motordrive, replace components of the motor drive, or the like. Theseadditional operations may be communicated to the operator 50 bysequentially modifying the visualization 84 at each instruction to mimicthe physical change to the motor drive that is to occur in thereal-world (e.g., non-virtual world).

FIGS. 5-12 depict various examples of graphical user interfacespresented by the operator device 54 via the display 70. To facilitatediscussion, operations of FIG. 4 are referred to in the followingdiscussion. It is noted that, as described herein, the operator device54 visually renders the image data of the AR UX workflow dataset forreference by the operator 50, however it should be understood that anycombination of indications may be used with the AR UX workflowvisualizations to provide guidance via the operator device 54 (e.g., toprovide AR UX workflow instructions). For example, the operator device54 may be used in conjunction with noise canceling headphones orearplugs, such that verbal instructions or audible cues may be presentedto the operator 50 via the AR UX workflow visualizations.

FIG. 5 is an illustration of a first example experience navigator (EN)application graphical user interface (GUI) 132A of the operator device54 after block 112 of the method 110. At block 112, the operator device54 may receive the request for AR support. In some cases, the operator50 may launch an application to initiate a request for AR support. Theoperator device 54 may receive the command to open the application andrespond by running an experience navigator application (e.g., executingcode that, when executed, provides the experience navigatorapplication). The experience navigator application may rendervisualizations, such that an experience navigator portal 134 is renderedon the display 70.

The experience navigator portal 134 may enable the operator 50 to inputquery parameters. In response to receiving the query parameters, theoperator device 54 may initiate a query of the AR UX system database 98to retrieve a subset of AR UX workflows datasets relevant to the queryparameters. The operator device 54 may present via its display 70indications corresponding to each of the subset of the AR UX workflowdatasets. Presentation of the indications may permit the operator 50 toselect between the subset of AR UX workflow datasets a respective AR UXworkflow instruction to download and/or render via the operator device54. In this way, when the AR UX workflow dataset is selected via aselection of the indication, the operator device 54 may initiate adownload of the selected AR UX workflow dataset. When at least a firstportion of the selected AR UX workflow dataset is downloaded to theoperator device 54, the operator device 54 may present an AR UX workflowvisualization corresponding to the first portion of the AR UX workflowdataset. As will be appreciated, such as during discussion of FIG. 6 ,the indications corresponding to the AR UX workflow datasets may includeselectable text describing the AR UX workflow dataset, selectable imagesdescribing the AR UX workflow dataset, or the like. Text or imagesdescribing the AR UX workflow dataset may correspond to a visualizationof a product associated with the AR UX workflow dataset (e.g., arepresentation of a virtual product of the AR UX workflow dataset), anoperation associated with AR UX workflow instructions of the AR UXworkflow dataset, or the like. In this way, when rendering theindications corresponding to the AR UX workflow datasets, the operatordevice 54 may refer to a subset of information obtained by the AR UXsystem 82 when performing the query of the AR UX system database 98,such as one or more virtual products of the AR UX workflow dataset or anexperience type corresponding to the AR UX workflow dataset.

To enable query parameter to be input, the experience navigator portal134 may include option selectors 136 (e.g., selector 136A, selector136B, selector 136C), such that each option selector 136 may include adrop-down menu storing a closed list of selectable options for theoperator 50. In some embodiments, the selectors 136 may be radiobuttons, input fields, or the like, to permit a variety of options forquerying. Additionally or alternatively, rendering of the experiencenavigator portal 134 may include rendering of menu buttons 138 to assistwith navigation through features of the experience navigator (EN)application. For example, menu button 138A may navigate the ENapplication GUI 132 to an operator profile and menu button 138B maynavigate the EN application GUI 132 to an application informationdocument that technically describes a related industrial automationdevice such as providing installation information, debugginginformation, components associated to the industrial automation device,or the like. The menu button 138C may navigate the EN application GUI132 to a home page (e.g., represented by EN application GUI 132A), menubutton 138D may navigate the EN application GUI 132 to a barcode (e.g.,a QR code, a matrix barcode) and/or a radio frequency identification tag(RFID tag) scanner, menu button 138E may navigate the EN application GUI132 to a customer service representative for support with performing anoperation according to a AR UX workflow instruction and/or associatedwith the application.

After the operator 50 enters inputs into the EN application GUI 132A viathe selectors 136 and initiates the query via search button 140, the ENapplication GUI 132A may update into an EN application GUI 132B. FIG. 6is an illustration of a second example EN application GUI 132B of theoperator device 54 after block 118 of the method 110. After block 118,the operator device 54 receives query results according to inputstransmitted to the AR UX system 82 (e.g., inputs received via selectors136). The query may result in multiple AR UX workflow datasets and thesemultiple results may be transmitted from the AR UX system 82 to theoperator device 54. In response to receiving the multiple query results,the operator device 54 may update the display 70 to present the ENapplication GUI 132B that includes indications of query results 150.Since the experience type was not specified in the query (e.g.,represented by selected selectors 152), multiple experiences for thesame product type and product are listed in the query results 150. Insome embodiments, a subset of experiences available to be selected viathe selector 136C may dynamically change based on the option selectedvia the selector 136A and/or the selector 136B. In this way, a firstproduct may have four experiences available and a second product mayhave two of the four experiences available. When the operator 50 selectsthe second product, the two of the four experiences may be presented asoptions for the selector 136C, such that the operator 50 is not giventhe option to select unavailable experiences for the second product.

FIGS. 7-12 depict additional features of the application discussed inFIG. 5 and FIG. 6 . For example, FIG. 7 is an illustration of a thirdexample EN application GUI 132C of the operator device 54. The ENapplication GUI 132C shows a library of AR UX workflow datasetsavailable for selection.

Workflows that are accessed relatively more frequently than otherworkflows by the operator device 54 may be featured via the ENapplication GUI 132C for convenient access. In some embodiments, the ARUX system 82 may transmit AR UX workflow datasets to the operator device54 in accordance with preventive maintenance schedules or according todetermined access patterns by the operator device 54. The preventivemaintenance schedules and/or the access patterns may be stored in theprofile database 94 according to industrial automation system 10 and/oroperator 50 profiles.

After the AR UX workflow dataset is selected, the operator device 54 mayinitiate the presentation of AR UX workflow visualizations. Initiationof the AR UX workflow visualizations may include placement of avisualization 84 of the product type associated with the selected AR UXworkflow dataset. FIG. 8 illustrates an example visualization presentedvia EN application GUI 132D. The EN application GUI 132D may begenerated by the operator device 54 in response to initiating a displayof the AR UX workflow visualizations at block 120. The AR UX workflowdataset may cause the operator device 54 to render animations of avisualization 84 of a virtual product that mimics operations of theprocedure (e.g., physical modifications, physical adjustments) to beperformed on the industrial automation device by the operator 50 in thereal-world. To do so, the operator 50 may virtually place the virtualproduct on a real-world surface (e.g., “tap to place”). In this depictedexample, the operator device 54 has received an AR UX workflow datasetassociated with a “Lower Fan Replacement” experience (e.g., operationalprocedure) and has generated a visualization viewable by the operator 50to guide the operator 50 through placing a virtual heatsink fan assemblyon the floor (e.g., a real-world surface) in the real-world environment.As such, the AR UX workflow dataset includes visualization data (e.g.,image data to be rendered) related to the selected workflow includingvisual representations of a device or component being maintained.

When the virtual product is placed, preliminary information may bedisplayed in a GUI by the operator device 54. FIG. 9 illustrates an ENapplication GUI 132E used by the operator device 54 to displaypreliminary information 162. Each AR UX workflow dataset may beassociated with a data file used to generate a visualization (e.g.,preliminary information 162) presented before animations associated withAR UX workflow instructions. Preliminary information 162 may includeinformation such as precautionary information to be read beforeperforming operations corresponding to AR UX workflow instructions,power down instructions to be performed before performing operationscorresponding to AR UX workflow instructions, a tool listingcorresponding to a list of equipment to be used when performingoperations corresponding to AR UX workflow instructions, a continuebutton to indicate that preliminary information 162 has been reviewed,or the like.

Preliminary information 162 may include buttons 164 (e.g., button 164A,button 164B, button 164C, button 164D) to enable the operator 50 tonavigate through information deemed useful for completing the AR UXworkflow instructions corresponding to a placed virtual product 166. Forexample, selection of the button 164A may cause the operator device 54to generate a GUI that indicates personal protective equipmentrequirements or other operational guidelines recommended to be followedwhile completing the AR UX workflow instructions. The button 164A maycause the operator device 54 to generate a GUI that indicates power downinstructions for a product corresponding to the virtual product 166. Forexample, FIG. 10 illustrates an EN application GUI 132F used by theoperator device 54 to display power down instructions 174 correspondingto the virtual product 166. In this example, the virtual product 166corresponds to a “Heatsink Fan Assembly.” It is noted that when othervirtual products are placed as part of initiation of different AR UXworkflow datasets, the visualization 84 used to represent the product asa virtual product as well as associated information (e.g., guidelinesgenerated via button 164A, power down instructions generated via button164B) may change.

Returning to FIG. 9 , the button 164C may cause the operator device 54to generate a GUI (e.g., an application GUI) that outlines tools to beused when completing the AR UX workflow instructions. For example, FIG.11 illustrates an EN application GUI 132G used by the operator device 54to display a tool listing 186 corresponding to AR UX workflowinstructions for the virtual product 166. Returning to FIG. 9 , thebutton 164D may cause the operator device 54 to generate a GUI thatguides the operator 50 to perform the desired experience. The button164D may be similar to a “continue” button or operation, where theoperator device 54 waits to begin showing an AR UX workflowvisualization of a first instruction until provided indication that thedevice to be operated on is in a no-power state or is otherwise ready tobe operated on.

FIG. 12 illustrates an EN application GUI 132H used by the operatordevice 54 to display the AR UX workflow visualization for the virtualproduct 166. The EN application GUI 132H may include a selector 198(e.g., a drop-down menu) that presents a list of operations associatedwith the AR UX workflow instructions to the operator 50 when selected.The list of operations indicated via the selector 198 may correspond toa list of operations specifically illustrated (e.g., highlighted,called-out) via the AR UX workflow visualizations. To start therendering of AR UX workflow visualizations corresponding to the AR UXworkflow instructions, the operator 50 may interact with a button 200.In response to receiving an input at the button 200, the operator device54 may render a first AR UX workflow visualization. In FIG. 12 , thevirtual product 166 is considered a dropped device. The virtual product166 may be a visualization 84 rendered as an overlaid three-dimensionalimage over an image data stream (e.g., live feed view of image data)received by the image sensor 66 of the operator device 54. As theoperator device 54 rotates around the virtual product 166, a view of thevirtual product 166 may update as well. In this sense, the virtualproduct 166 is locked in place after the device is virtually placed intothe real-world view of the AR lens of the operator device 54.

To elaborate further on operations of the operator device 54 forpresenting the AR UX workflow visualizations, FIG. 13 is a flowchart ofa method 210 performed by the operator device 54 to receive and displaythe AR UX workflow visualizations. Although the method 210 is describedbelow as performed by the operator device 54, it should be noted thatthe method 210 may be performed by any suitable processor that presentsAR UX workflow visualizations. Moreover, although the followingdescription of the method 210 is described in a particular order, itshould be noted that the method 210 may be performed in any suitableorder.

At block 212, the operator device 54 may display AR image data of an ARUX workflow dataset. The operator device 54 may receive the AR UXworkflow dataset at block 120 of method 110 from the AR UX system 82. ARimage data may correspond to a first portion of the AR UX workflowdataset to be presented by the operator device 54 on the display 70and/or may correspond to each portion of the AR UX workflow dataset, andthus include the data corresponding to the first portion. When the ARimage data corresponds to the first portion, the operator device 54 mayreceive additional portions of AR image data that correspond tosubsequent portions of the AR UX workflow dataset at a later time butbefore presenting the additional portions. Displaying the AR image datamay include displaying the AR image data corresponding to the firstportion of the AR UX workflow dataset. The image rendered on the display70 in response to presenting the AR image data may correspond to the ENapplication GUI 132D. As shown in the EN application GUI 132D, a portionof the AR image data may be rendered images that are static while aportion of the AR image data may correspond to a live-feed (e.g., realtime video captured from the image sensor 66 of the operator device 54)of the ambient environment of the operator device 54. The operator 50may have the option to physically move the operator device 54 around inthe real-world to align virtual objects (e.g., virtual product 166,text) generated using the AR image data suitably with physical objectsof the real-world.

For example, at block 214, the operator device 54 presenting the ENapplication GUI 132D may detect when a portion of a virtual objectcorresponding to the AR image data is aligned with an industrialautomation device 20 of the real-world. The operator device 54 maydetect the alignment when the portion of AR image data for the virtualobject is aligned with a portion of data acquired via the image sensor66 determined to correspond to the industrial automation device 20. Whenthe virtual object is aligned with the industrial automation device 20,the AR environment may lock the virtual object to the industrialautomation device 20.

In some embodiments, the virtual product 166 may be virtually positionedin the AR environment to overlay the physical industrial automationdevice 20 in the real-world (e.g., as seen via the AR environment). Whenthis occurs, the virtual product 166 may, at block 216, be locked by theoperator device 54 to the industrial automation device 20 in the ARenvironment. The operator device 54 may also automatically lock thevirtual product 166 to the real-life industrial automation device 20.For example, the operator device 54 may use image recognition processesto detect where the specified industrial automation device 20 is withinthe real-time video stream and operate to render the virtual product 166as overlaid or snapped to the industrial automation device 20. In someembodiments, image recognition techniques may be used to detect when thevirtual object matches a real object. The detection may cause theoperator device 54 to lock the virtual object to the real object.

When the virtual product 166 is locked to the industrial automationdevice 20, changes to the industrial automation device 20 in thereal-world may be detected by the operator device 54 via the experiencenavigator application. The operator device 54 detecting changes to theindustrial automation device 20 may enable the operator device 54 tonotify the operator 50 when an operation was incorrectly performed(e.g., the wrong component being adjusted, an incorrect orientation of acomponent) and/or may identify when an operation was performed correctlysuch that the AR UX workflow may automatically progress to a subsequentAR UX workflow instruction. In response to detecting that an operationwas incorrectly performed, the operator device 54 and/or the AR UXsystem 82 may adjust a color scheme of the AR UX workflow visualizationto bring attention to the incorrect operation (e.g., change the color ofthe AR UX workflow visualization from a native color scheme of imagedata corresponding to the surrounding environment to a relatively highercontrast and/or a black and white color scheme). This change in colorscheme may also occur when the operator device 54 detects that it hasbeen moved into a barricade or moved outside a boundary virtuallyestablished for presentation and/or interaction with the AR UX workflowvisualization.

Furthermore, when the virtual product 166 is locked to the industrialautomation device 20, the virtual product 166 and the industrialautomation device 20 may rotate similarly as the operator device 54 ismoved around the industrial automation device 20 in the real-world. Forexample, the operator device 54 may initially be positioned to capture,via the image sensor 66, a first side of the industrial automationdevice 20 in the real-world and be rotated in the real-world to capturea second side of industrial automation device 20. When the operatordevice 54 is rotated to see the second side of the industrial automationdevice 20, the virtual product 166 tracks the rotation the industrialautomation device 20 within the AR environment such that the second sideof the virtual product is shown as overlaid to the second side of theindustrial automation device 20.

In response to any locking of the virtual product 166 and/or anyplacement of virtual objects of the AR UX workflow dataset, at block218, the operator device 54 may present AR UX workflow options. Theoptions may correspond to additional information deemed relativelyuseful to include with operational instructions to be presented duringthe AR UX workflow. An example of AR UX workflow options may be shownvia the EN application GUI 132E in FIG. 9 . In the EN application GUI132E, AR UX workflow options are presented as the preliminaryinformation 162 which the operator 50 may use to navigate through theadditional information for the AR UX workflow dataset.

When the operator device 54 receives an input corresponding to adepicted selectable option, the operator device may, at block 222,present information associated with the selection. For instance, thebutton 164D of EN application GUI 132E in FIG. 9 is associated withinitiation of the AR UX workflow visualizations. After receiving thisinput, the operator device 54 may render a portion of the AR UX workflowvisualizations corresponding to a first operation that the operator 50is to perform as part of the AR UX workflow. In some embodiments, the ARUX workflow instructions do not immediately start at the firstoperation. For example, FIG. 12 depicted the selector 198 (e.g., adrop-down menu) that the operator 50 may use to start the first AR UXworkflow instruction (e.g., by selecting the first operation from thedrop-down list within the drop-down menu).

When the first operation is initiated (e.g., automatically, in responseto an input at the selector 198), the operator device 54 may, at block224, animate a portion of the AR UX workflow visualization correspondingto the first operation. The animation presented may mimic and show theoperator 50 what physical manipulations to perform to the industrialautomation device 20. For example, FIG. 14 illustrates an EN applicationGUI 132I used by the operator device 54 to animate an example operationof an example AR UX workflow. In FIG. 14 , the example operationcorresponds to “1-Remove Front Screen,” captured in the selector 198.The virtual product 166, “Heatsink Fan Assembly,” is to have its frontscreen removed as a result of the example operation of the AR UXworkflow instruction. Thus, images rendered by the operator device 54may guide the operator 50 through the operation. The images may beanimated to improve how the operation is described to the operator 50and sometimes the animations may start in response to an input beingreceived via interaction with the button 200 (e.g., operator 50 pressingthe button 200). Furthermore, additional product information 242 may beoverlaid on images rendered that correspond to the ambient real-world.The additional product information 242 may provide details that might beuseful to the operator 50 while performing the animated operation.

To elaborate on the animation, FIG. 15 illustrates an EN application GUI132I used by the operator device 54 that animates the example operationdescribed in FIG. 14 . Comparing FIG. 14 to FIG. 15 , the animationcorresponding to the operation of removing the front screen from theheatsink fan assembly may result in the virtual product 166 beingrendered by the operator device 54 as having the front screen partiallyremoved to demonstrate the operation to be performed by the operator 50.Furthermore, emphasis graphics 254 may be used to highlight and drawattention to a portion of the virtual product 166 that is to bemanipulated in the real-world through the current animation of the AR UXworkflow visualization. In some embodiments, the emphasis graphics 254may also be used to outline and/or define regions within which theoperator 50 is permitted or recommended to move the operator device 54around within to access the AR UX workflow dataset. These outlinedand/or defined regions may correspond to operational zones within whichadditional personal protective equipment or additional precaution doesnot need to be taken by the operator 50. The operator 50, in response toviewing the animation represented by FIG. 14 and FIG. 15 , mayunderstand better which portion of the industrial automation device 20to adjust per the AR UX workflow instruction.

Returning to FIG. 13 , the operator device 54 may, at block 226, receiveupdated image data of the industrial automation device 20 from the imagesensor 66, and use the updated image data, at block 226, to verifywhether the operation instructed via the animation was suitablyperformed in the real-world and thus whether the AR UX workflowinstruction was completed. For example, the operator device 54 mayanalyze the updated image data to detect whether the front screen wasremoved from industrial automation device 20. The operator device 54 mayperform operations based on machine learning operations, artificialintelligence operations, or other suitable operations that may detectwhen an operation is suitably performed according to the AR UX workflowinstruction. When the operator device 54 determines that the operationinstructed is not completed, the operator device 54 may repeat analysisof captured image data at block 226 to determine when the operation iscompleted. In some cases, the operator device 54 may determine theoperation was not completed as instructed and was indeed incorrectlyperformed (e.g., a part was installed upside, the wrong component wasinstalled). In these cases, the operator device 54 may generate anotification to alert the operator 50 of the incorrect operation. Thismay help to reduce a likelihood of mis-operation propagating to furtheractivities of the operator 50.

When the operator device 54 determines that the operation instructed atblock 224 is completed, the operator device 54 may, at block 230,determine whether another portion to the AR UX workflow dataset isavailable for presentation via the display 70. The operator device 54may reference the memory 60 or the storage 62 to retrieve any additionalAR UX workflow data (e.g., AR image data) and/or may request additionalAR UX workflow data (e.g., AR image data corresponding to a next portionof the AR UX workflow dataset) from the AR UX system 82.

When additional AR UX workflow data is available, the operator devicemay, at block 232, render and animate the subsequent AR UX workflowinstruction. The additional AR UX workflow data may correspond to asecond operation instructing a second adjustment to the industrialautomation device 20 to be performed after the first operationinstructing a first adjustment to the industrial automation device 20.This process of checking for and retrieving subsequent portions of theAR UX workflow dataset permits the operator device 54 to progressthrough the AR UX workflow instructions, and thus progress throughproviding instructions for the corresponding procedure (e.g., experiencetype) to the operator 50.

FIG. 16 illustrates an EN application GUI 132K used by the operatordevice 54 to animate a second example operation subsequent to theexample operation described in FIG. 14 . When the subsequent AR UXworkflow visualization is rendered, the selector 198 may be visualizedas advanced to a next option corresponding to the subsequent operation.In this example, the subsequent operation corresponds to “2-Disconnectwires.” Similar to additional product information 242, additionalproduct information 266 associated with disconnecting wires is presentedvia the EN application GUI 132K. Furthermore, the operator device 54adjusted the rendering of the emphasis graphics 254 to emphasize aportion of the virtual product that corresponds to the subsequentoperation (e.g., wires to be disconnected).

Returning to FIG. 13 , when the subsequent portion is animated, theoperator device 54 may, at block 226, receive updated image data of theindustrial automation device 20 and verify that operations correspondingto the subsequent portion of the AR UX workflow instructions weresuitably performed to the industrial automation device 20. For theexample operation of FIG. 16 , the operator device 54 may verify thatthe wires highlighted via emphasis graphics 254 were indeeddisconnected. The operator device 54 may analyze the updated image datato detect whether the wires were disconnected. When the operator device54 determines that the operations were performed in accordance withinstructions presented at block 232, the operator device 54 may repeatthe determination of whether there are subsequent portions of the AR UXworkflow instructions to be displayed and whether any subsequentportions of the AR UX workflow instructions were performed by theoperator.

In response to determining that each portion of the AR UX workflowinstructions was correctly performed by the operator 50 to theindustrial automation device 20, the operator device 54 may proceed toend presentation of the AR UX workflow instructions and, at block 234,update a profile corresponding to the operator device 54. The profileupdated by the operator device 54 may indicate a most recent maintenanceoperation performed by the operator 50 corresponding to the operatordevice 54. The profile update may additionally or alternatively update amaintenance history corresponding to the industrial automation device20. The maintenance history may indicate a time that the industrialautomation device 20 was serviced by an operator (e.g., operator 50)and/or the specific operation that was performed to the industrialautomation device 20 (e.g., component replacement, preventivemaintenance, lock-out/tag-out operation). The maintenance history may bereferenced by other industrial automation devices 20, control systems,or the like to determine whether the industrial automation device 20 isdue for servicing (e.g., mechanical or electrical maintenance work) orthe like.

As described above, the operator device 54 may render the selector 198on the display 70 when providing the AR UX workflow visualizations. Insome embodiments, the selector 198 may be used to navigate to differentportions of the AR UX workflow instructions (e.g., between differentanimations of the AR UX workflow visualizations).

As an example, FIG. 17 illustrates an EN application GUI 132L used bythe operator device 54 after receiving an indication to present eachportion option of the AR UX workflow visualization via the selector 198.When the operator device 54 receives a selection at the selector 198(e.g., operator 50 pressing the selector 198), the operator device 54may render a selection menu 278 that outlines each of the operations ofthe AR UX workflow instructions (e.g., represented by indications 280).The operator device 54 may render the animation and/or portion of the ARUX workflow visualization corresponding to the particular operation whenthe particular operation is selected from the selection menu 278. Asdifferent operations are selected from the selection menu 278, thevirtual product 166 may change renderings to match a final frame of ananimation corresponding to a previous portion (e.g., one animatedportion before the selected animated portion). For example, if theoperation corresponding to “3-Remove thinlet seal mounting plate” wasselected from the selection menu 278, the operator device 54 may updatea rendering of the virtual product 166 to reflect what the virtualproduct 166 was previously rendered as at the end of the animationcorresponding to “2-Disconnect wires.”

To elaborate on workflow animations, FIGS. 18-21 are illustrationscorresponding to an example animation corresponding to operations for“3-Remove thinlet seal mounting plate.” In particular, FIG. 18illustrates an EN application GUI 132M of a subsequent step (e.g., asecond step) after the EN application GUI 132N of FIG. 17 , FIG. 19illustrates an EN application GUI 132N of a third step of the AR UXworkflow instructions, FIG. 20 illustrates an EN application GUI 132O ofa fourth step of the AR UX workflow instructions, and FIG. 21illustrates an EN application GUI 132P of a fifth step of the AR UXworkflow instructions. For ease of description, FIGS. 18-21 aredescribed together below.

The virtual product 166 may remain locked during the animationcorresponding to “3-Remove thinlet seal mounting plate.” In this way,the operator 50 may move the operator device 54 around and within aphysical space nearby to the industrial automation device 20 whilewatching the AR UX workflow visualization. For example, the virtualproduct 166 remains at a same location within the AR environment as theoperator device 54 is moved between views presented in FIG. 18 and FIG.19 . The animation of the AR UX workflow visualization may movecomponents of the virtual product 166 to emphasize or highlight anoperation to be performed to the industrial automation device 20. Forexample, a thinlet seal mounting plate 292 may be animated as beingremoved from the virtual product 166 via the sequence of the applicationGUIs 132M, 132N, 1320, 132P (e.g., FIGS. 18-21 ). The animation maycause the thinlet seal mounting plate 292 rendered in a first position(e.g., as shown in FIG. 18 ) to be rendered in a second position by theend of the animation (e.g., as shown in FIG. 21 ). It is noted that theemphasis graphics 254 may move during the animation and/or may remainlocked relative to the virtual product 166 during the animation. In thedepicted example animation, the emphasis graphics 254 remained locked tothe virtual product 166 during the course of the animation.

At the end of presenting each of the AR UX workflow instructions, theoperator device 54 may render an exit button visualization. For example,FIG. 22 illustrates an EN application GUI 132Q of a sixth step of the ARUX workflow instructions. The EN application GUI 132Q includes avisualization of a stop button 304. The portion of the AR UX workflowinstructions corresponding to “4-Remove Fan Assembly” may be at leastpartially depicted using the EN application GUI 132Q and may represent alast portion of the AR UX workflow instructions. The EN application GUI132Q may correspond to a last frame of the animation. Since theanimation is completed and the portion depicted is the last portion ofthe AR UX workflow instructions, the operator device 54 may render thestop button 304 to enable exit from the AR UX workflow instructioncorresponding to the “Heatsink Fan Assembly.” The operator device 54 mayreceive an input via the stop button 304 and, in response to the input,end presentation of the AR UX workflow visualizations.

In some embodiments, the operator 50 may request additional supportwhile performing operations corresponding to AR UX workflow instructionson the industrial automation device 20. For example, the operator 50 maybe unable to determine where a particular component is located, or whichcomponent to remove, or the like. To reduce a likelihood ofmis-operation, the operator device 54 may render the menu button 138E(e.g., described in FIG. 5 ) to permit the operator 50 convenient accessto a customer service representative. When the menu button 138E receivesan input, the operator device 54 may render a communication portal topermit the customer service representative and the operator 50 tointercommunicate in real-time. The operator device 54 may additionallyor alternatively telephonically connect the customer servicerepresentative and the operator 50 (e.g., via a voice chat or telephonecall). In some cases, the customer service representative may use acomputing device to transmit a modified AR UX workflow visualization tothe operator device 54 as a way to help elaborate on a portion of the ARUX workflow instruction indicated by the operator 50 to be confusing.

To help elaborate, FIG. 23 is a block diagram of the operator device 54receiving a modified AR UX workflow visualization from a customerservice representative (CSR) device 316 associated with a customerservice representative 318. Similar to other electronic devicesdescribed herein, the CSR device 316 may be any suitable computingdevice, such as a personal computing device, a laptop, a desktop, asmart phone, a cellular device, a tablet, a personal wearable device, orthe like. The operator device 54 and the CSR device 316 may becommunicatively coupled via the network 90. The operator device 54, inresponse to the menu button 138E receiving an input, may transmit theongoing AR UX workflow dataset to the CSR device 316 (e.g., such astransmitting an actively rendered portion of the AR UX workflow datasetto the CSR device 316). In some embodiments, the operator device 54 maynotify the AR UX system 82 associated with the menu button 138Ereceiving the input and cause the AR UX system 82 to provision an AR UXworkflow visualization to both the CSR device 316 and the operatordevice 54 and/or to provision the ongoing AR UX workflow visualizationto the CSR device 316 on behalf of the operator device 54. The AR UXsystem 82 may provide the AR UX workflow visualization to the CSR device316 from the AR UX system database 98.

When the CSR device 316 receives data for the AR UX workflowvisualization, the CSR device 316 may render at least a portion of theAR UX workflow visualization on a display 70, such that the customerservice representative 318 may follow along as the operator 50 interactswith the AR UX workflow visualization rendered by the operator device54. In some embodiments, the CSR device 316 and the operator device 54may pass control of the AR UX workflow visualization between the CSRdevice 316 and the operator device 54, such that not more than onedevice is able to affect the AR UX workflow visualization at a time. Inthese cases, changes made via the CSR device 316 or the operator device54 may be rendered automatically in real-time by the other of the CSRdevice 316 or the operator device 54.

For example, the CSR device 316 may receive an input and convert theinput into modifications overlaid on the AR UX workflow visualizationand/or the AR environment. The input may be converted into overlaidtext, overlaid chalk modifications, overlaid clipart or other geometricrenderings, or other suitable visualizations to emphasize a portion ofthe AR UX workflow visualization.

To help illustrate, FIG. 24 illustrates an EN application GUI 132Rdepicting user graphic modifications overlaid on an AR UX workflowvisualization being presented on the operator device 54 that may berendered substantially simultaneously on the CSR device 316 and theoperator device 54. In this way, when the CSR device 316 receives aninput and generates a chalk modification 326 in response to the input,the operator device 54 may receive the chalk modification 326 inreal-time. The chalk modification 326 may be similar to the emphasisgraphics 254 that draw attention to a particular portion of renderedreal-time real-world images captured by the image sensor 66 of theoperator device 54. For example, the operator 50 may ask the customerservice representative 318 a question, such as, “which wire do Idisconnect?” The customer service representative 318, in addition to anyverbal response, may also provide a visual response via the chalkmodification 326 to further elaborate on the point to be expressed tothe operator 50. The operator device 54 and the CSR device 316 mayintercommunication in a variety of suitable ways including text or voicecommunication natively supported by the type of device, such as when theoperator device 54 and the CSR device 316 are cellular-capableelectronic devices, communication via text messages (e.g., SMS, RMS), orthrough phone calls, may be permitted. In some embodiments, the operatordevice 54 and the CSR device 316 may communicate by sending voicerecording via the AR UX system 82, by exchanging text, lines, clipart,shapes, or the like. Sometimes the text may include hyperlinks or linksto information for reference while performing the AR UX workflowinstruction. For example, the CSR device 316 may transmit a hyperlink toan additional workflow dataset to assist with performing the current ARUX workflow instruction.

The customer service representative 318 may freely move the CSR device316 to interact with its AR environment independent of the ARenvironment or relative location of the operator 54. When anymodifications generated by the CSR device 316 and rendered on the AR UXworkflow visualization of the operator device 54 are disposed in aportion of the AR environment outside of a portion of the AR environmentrendered on the operator device 54, the operator device 54 may generateicons and/or may use haptic feedback to guide the operator 50 toposition the operator device 54 within the real-world. For example, acustomer service representative 318 may place haptic feedback into theAR environment as associated with a portion of the AR UX workflowvisualization to help guide the operator 50 to rotate the operatordevice 54 toward the placed haptic feedback indicator. The AR UX system82 may receive a haptic feedback placement from the CSR device 316indicative of a relative placement of the haptic feedback into the ARenvironment generated via the CSR device 316. The AR UX system 82 mayupdate a portion of the AR UX workflow dataset and transmit the updatedportion to the operator device 54. The operator device 54, in responseto receiving the updated portion of the AR UX workflow dataset, mayrender the updated portion of the AR UX workflow dataset. Rendering theupdated portion may cause the operator device 54 to provide directionalhaptic feedback to guide a rotation of the image sensor 66 toward thelocation indicated via the haptic feedback placement.

In some embodiments, the CSR device 316 may provide audio feedbackplacement to the AR UX system 82 to communicate directionality viapresentation of the AR UX visualization by the operator device 54. Insome cases, the AR UX visualization may automatically generate feedbackto be provided via the operator device 54 to guide the operator 50 to aregion of the AR UX workflow visualization indicated via the CSR device316. For example, the AR UX system 82 may receive image data from theoperator device 54 and may determine a first directionality of aposition in the real-world of the operator device 54 based at least inpart on the image data (e.g., may determine a direction in which theoperator device 54 is orientated in based on what is captured via theimage data). The AR UX system 82 may determine a second directionalityassociated with an input into the image data rendered on the display 70of the CSR device 316 (e.g., to determine where the customer servicerepresentative 318 is instructing the operator 50 to view). The AR UXsystem 82 may generate an indication of feedback (e.g., haptic feedback,audio feedback) based at least in part on a difference between the firstdirectionality and the second directionality, such as to guide arotation or physical reorientation of the operator device 54 in thedirection indicated by the feedback. The AR UX system 82 may transmitthe indication of feedback to the operator device 54, which, inresponse, provides a directional indication (e.g., a directional noise,a directional vibration). The specific direction indicated by thedirectional indication, a pulse frequency, a pulse intensity, an audiofile, an audible sound frequency, or the like, may be defined by datatransmitted from the AR UX system 82 to the operator device 54 as partof the indication of feedback.

In some embodiments, the CSR device 316 may render two or more AR UXworkflows visualizations for viewing and/or manipulation by the customerservice representative 318. For example, the CSR device 316 may receiveand render a first AR UX workflow visualization corresponding to thelive video stream recorded via the operator device 54 and may receiveand render a second AR UX workflow visualization corresponding to a copyAR environment locally placed by the CSR device 316. The customerservice representative 318 may use the first AR UX workflowvisualization to determine what the operator 50 is looking at orinteracting with and may use the second AR UX workflow visualization toprovide guidance to the operator 50. For example, the operator 50 mayuse the operator device 54 to drop an example virtual product 166 andinitiate a corresponding AR UX workflow visualization, and in responseto connecting with the operator device 54, the customer servicerepresentative may drop the example virtual product 166 and initiate acorresponding AR UX workflow visualization in a surrounding environmentof the CSR device 316. In this way, the CSR device 316 may modify the ARUX workflow visualization for the example virtual product 166 in the ARenvironment corresponding to the CSR device 316 and just themodifications may be rendered on the AR UX workflow visualization of theoperator device 54.

For example, the operator device 54 may display image data on itsdisplay 70. The CSR device 316 may display the image data displayed bythe operator device 54 on a first portion of its display 70 and maydisplay a locally maintained AR UX workflow visualization on a secondportion of its display. When the operator device 54 updates itsdisplayed image data, the CSR device 316 may mirror the update and alsoupdate the first portion of its display 70. In some cases, manipulationsto the second portion of the display 70 of the CSR device 316 may beused to generate notifications, alerts, overlaid images, or the like tobe presented via the display 70 of the operator device 54. In this way,the AR UX system 82 may receive an input from the CSR device 316corresponding to a manipulation of the second portion of the display 70of the CSR device 316. The AR UX system 82 may generate an adjusted ARUX workflow dataset based at least in part on the input from the CSRdevice 316 and may transmit the adjusted AR UX workflow dataset to theoperator device 54 and to the CSR device 316. The operator device 54 mayupdate its display 70 with a representation of the manipulation made tothe second portion of the display 70 of the CSR device 316 in responseto receiving the adjusted AR UX workflow dataset. In some cases, the CSRdevice 316 may update the first portion of its display 70 with therepresentation of the manipulation in response to receiving the adjustedAR UX workflow dataset from the AR UX system 82, such that presentationof the AR UX workflow visualizations is mirrored between the display 70of the operator device 54 and the display 70 of the CSR device 316. Themanipulation may include any suitable change or adjustment to the AR UXworkflow visualization including, for example, audio feedback placement,haptic feedback placement, one or more lines, text, one or more overlaidimages, or the like.

In some embodiments, the CSR device 316 may display the AR UX workflowvisualizations at a delayed rate relative to displaying operations ofthe operator device 54 (or vice versa). In these cases, the AR UX system82 may compensate for the processing and/or displaying delays bystaggering transmission times of the portions of the AR UX workflowdatasets to respective of the CSR device 316 and/or the operator device54. For example, when the CSR device 316 displays AR UX workflowvisualizations corresponding to the AR UX workflow dataset a duration oftime before the operator device 54 displays the same AR UX workflowvisualizations, the AR UX system 82 may transmit the AR UX workflowdataset to the operator device 54 at a leading time equal to theduration of time (e.g., at a time selected to compensate for theduration of time difference between displaying operations), such thatthe AR UX workflow visualizations are presented simultaneously on boththe CSR device 316 and the operator device 54.

Keeping the foregoing in mind, in some embodiments, the AR UX system 82may filter a subset of AR UX workflows visualization from the AR UXsystem database 98 before querying the AR UX system database 98 to makethe query results relatively more relevant to the operator device 54. Asdescribed above, sometimes filtering is performed based at least in parton information stored in the location anchors database 96. Usinglocation information to perform filtering may enable the AR UX system 82to query a subset of AR UX workflow instructions that are associatedwith industrial automation devices 20 that are located closer or withina threshold distance from the operator device 54.

Technical effects of the present disclosure include techniques forfacilitating the operation of an industrial automation system by anoperator. An augmented reality (AR) environment may be provided via anoperator device that provides animated instructions for the operator toreference when performing procedures to industrial automation devices ofthe industrial automation system. The operator device may render a firstportion of an AR user experience (UX) workflow visualization, determinewhen an operation corresponding to the first portion of the AR UXworkflow instruction is performed, and in response to the first portionbeing performed, may render a second portion of the AR UX workflowvisualization subsequent to the first portion of the AR UX workflowvisualization. In some embodiments, the operator device may communicatewith a customer service representative device (CSR device) while the ARUX workflow visualization is being presented. The CSR device may provideinputs to be visually overlaid on the AR environment and a renderedportion of the AR UX workflow visualization as a way to presentadditional guidance to the operator using the operator device. Using ARUX workflows may improve operations of the industrial automation systemby improving systems and methods used to provide procedural instructionsto operators. As discussed herein, additional systems and methods may beused to improve the provision of AR UX workflow visualizations to theoperator device, such as filtering an AR UX system database to bequeried to reduce an amount of time to identify AR UX workflowvisualizations to be rendered via the operator device.

While only certain features of the disclosure have been illustrated anddescribed herein, many modifications and changes will occur to thoseskilled in the art. It is, therefore, to be understood that the appendedclaims are intended to cover all such modifications and changes as fallwithin the true spirit of the disclosure.

What is claimed is:
 1. A system, comprising: a processor; and memorycomprising computer-executable code that, when executed by theprocessor, causes the processor to: receive a service request from afirst device, wherein the service request is configured to cause acomputing system to communicatively couple the first device to a seconddevice; transmit a workflow dataset associated with an industrialautomation device to the first device and the second device, wherein theworkflow dataset comprises: one or more instructions associated with oneor more operations for the industrial automation device; and one or morevirtual objects associated with the one or more instructions and theindustrial automation device, wherein the one or more virtual objectsare configured to: be rendered at a first position and orientation in afirst augmented environment of first image data depicted on a firstdisplay of the first device, wherein the first position and orientationare relative to a first location of the industrial automation device ina real-world environment; and be rendered at a second position andorientation in a second augmented environment of second image datadepicted on a second display of the second device, wherein the secondposition and orientation are relative to the first location of theindustrial automation device in the real-world environment; wherein theworkflow dataset is configured to cause the first device and the seconddevice to display an animation of a first virtual object of the one ormore virtual objects on the first image data depicted on the firstdisplay and the second image data depicted on the second display,respectively, and wherein the animation corresponds to performing afirst instruction of the one or more instructions; receive an input tothe second image data representative of the animation from the seconddevice, wherein the input comprises one or more visualizations at athird position and orientation in the second augmented environment;determine a fourth position and orientation within the first augmentedenvironment based on the third position and orientation and the firstposition and orientation; generate an indication of directional hapticfeedback based on the fourth position and orientation, wherein theindication of directional haptic feedback comprises a pulse intensityconfigured to convey a directional notification for a user of the firstdevice, wherein the directional notification is indicative of adirection relative to the user; transmit the one or more visualizationsto the first device for display via the first display; transmit theindication of directional haptic feedback to the first device, whereinthe indication of the directional haptic feedback is configured to causethe first device to output the directional haptic feedback using thepulse intensity, wherein the first device is configured to: receivethird image data representative of the industrial automation device froman image sensor associated with the first device; determine that thefirst instruction of the one or more instructions is not performed basedon a comparison between the third image data and the animationcorresponding to first virtual object performing the first instruction;and present a notification indicative of an incorrect performance of thefirst instruction via the first display in response to determining thatthe first instruction was not performed.
 2. The system of claim 1,wherein the one or more visualizations comprise one or more lines. 3.The system of claim 1, wherein the computer-executable code that, whenexecuted by the processor, causes the processor to: receive additionalimage data from the first device corresponding to a changed position ofthe first device; and transmit the additional image data to the seconddevice, wherein the second device is configured to render an updatedimage of the industrial automation device in response to receiving theadditional image data.
 4. The system of claim 1, wherein the one or morevisualizations comprise one or more overlaid shapes.
 5. The system ofclaim 1, wherein the one or more visualizations comprise text.
 6. Thesystem of claim 1, wherein the input comprises an audio recordingcorresponding to the one or more visualizations.
 7. The system of claim1, wherein the computer-executable code that, when executed by theprocessor, causes the processor to: receive an additional input from thesecond device, wherein the additional input comprises an audio feedbackrelative to the second image data; and transmit the additional input tothe first device for presentation via the first device, wherein theadditional input is configured to cause the first device to provide adirectional sound.
 8. A method, comprising: receiving, via a processor,a service request from a first device, wherein the service request isconfigured to cause a computing system to communicatively couple thefirst device to a second device; transmitting, via the processor, aworkflow dataset associated with an industrial automation device to thefirst device and the second device, wherein the workflow datasetcomprises: one or more instructions associated with one or moreoperations for the industrial automation device; and one or more virtualobjects associated with the one or more instructions and the industrialautomation device, wherein the one or more virtual objects areconfigured to: be rendered at a first position and orientation in afirst augmented environment of first image data depicted on a firstdisplay of the first device, wherein the first position and orientationis relative to a first location of the industrial automation device in areal-world environment; and be rendered at a second position andorientation in a second augmented environment of second image datadepicted on a second display of the second device, wherein the secondposition and orientation is relative to the first location of theindustrial automation device in the real-world environment; wherein theworkflow dataset is configured to cause the first device and the seconddevice to display an animation of a first virtual object of the one ormore virtual objects on the first image data depicted on the firstdisplay and the second image data depicted on the second display,respectively, and wherein the animation corresponds to performing afirst instruction of the one or more instructions; receiving, via theprocessor, a first input to the second image data representative of theanimation from the second device, wherein the first input comprises oneor more visualizations at a third position and orientation in the secondaugmented environment; determining, via the processor, a fourth positionand orientation within the first augmented environment based on thethird position and orientation and the first position and orientation;generating, via the processor, an indication of directional hapticfeedback based on the fourth position and orientation, wherein theindication of directional haptic feedback comprises a pulse intensityconfigured to convey a directional notification for a user of the firstdevice, wherein the directional notification is indicative of adirection relative to the user; transmitting, via the processor, the oneor more visualizations to the first device for display via the firstdisplay; and transmitting, via the processor, the indication ofdirectional haptic feedback to the first device, wherein the indicationof the directional haptic feedback is configured to cause the firstdevice to output the directional haptic feedback using the pulseintensity, wherein the first device is configured to: receive thirdimage data representative of the industrial automation device from animage sensor associated with the first device; determine that the firstinstruction of the one or more instructions is not performed based on acomparison between the third image data and the animation correspondingto first virtual object performing the first instruction; and present anotification indicative of an incorrect performance of the firstinstruction via the first display in response to determining that thefirst instruction was not performed.
 9. The method of claim 8,comprising: receiving a text input from the second device, wherein thetext input provides a link to an additional workflow dataset; andtransmitting the text input to the first device, wherein the firstdevice, in response to accessing the link to the additional workflowdataset, is configured to display an additional visualization associatedwith a portion of the additional workflow dataset.
 10. The method ofclaim 8, wherein the one or more visualizations comprise one or morelines.
 11. The method of claim 10, comprising receiving a second inputto the first image data from the first device after transmitting the oneor more visualizations to the first device, wherein the second input tothe first image data from the first device comprises one or moreadditional lines, one or more overlaid shapes, text, or any combinationthereof.
 12. The method of claim 8, comprising: receiving a first audioinput from the second device; transmitting the first audio input fromthe second device to the first device; and receiving a second audioinput from the first device after the first device receive the firstaudio input from the second device.
 13. A tangible, non-transitorycomputer-readable medium configured to store computer-executable codeexecutable by a processor of a computing device that, when executed bythe processor, causes the computing device to perform operationscomprising: receiving a service request from a first device, wherein theservice request is configured to cause a computing system tocommunicatively couple the first device to a second device; transmittinga workflow dataset associated with an industrial automation device tothe first device and the second device, wherein the workflow datasetcomprises: one or more instructions associated with one or moreoperations for the industrial automation device; and one or more virtualobjects associated with the one or more instructions and the industrialautomation device, wherein the one or more virtual objects areconfigured to: be rendered at a first position and orientation in afirst augmented environment of first image data depicted on a firstdisplay of the first device, wherein the first position and orientationis relative to a first location of the industrial automation device in areal-world environment; and be rendered at a second position andorientation in a second augmented environment of second image datadepicted on a second display of the second device, wherein the secondposition and orientation is relative to the first location of theindustrial automation device in the real-world environment; wherein theworkflow dataset is configured to cause the first device and the seconddevice to display an animation of a first virtual object of the one ormore virtual objects on the first image data depicted on the firstdisplay and the second image data depicted on the second display,respectively, and wherein the animation corresponds to performing afirst instruction of the one or more instructions; receiving a firstinput to the second image data representative of the animation from thesecond device, wherein the first input comprises one or morevisualizations at a third position and orientation in the secondaugmented environment; determining a fourth position and orientationwithin the first augmented environment based on the third position andorientation and the first position and orientation; generating anindication of directional haptic feedback based on the fourth positionand orientation, wherein the indication of directional haptic feedbackcomprises a pulse intensity configured to convey a directionalnotification for a user of the first device, wherein the directionalnotification is indicative of a direction relative to the user;transmitting the one or more visualizations to the first device fordisplay via the first display; and transmitting the indication ofdirectional haptic feedback to the first device, wherein the indicationof the directional haptic feedback is configured to cause the firstdevice to output the directional haptic feedback using the pulseintensity, wherein the first device is configured to: receive thirdimage data representative of the industrial automation device from animage sensor associated with the first device; determine that the firstinstruction of the one or more instructions is not performed based on acomparison between the third image data and the animation correspondingto first virtual object performing the first instruction; and present anotification indicative of an incorrect performance of the firstinstruction via the first display in response to determining that thefirst instruction was not performed.
 14. The non-transitorycomputer-readable medium of claim 13, wherein the workflow dataset isconfigured to cause the first device and the second device to displaythe animation at a same time.
 15. The non-transitory computer-readablemedium of claim 13, the operations comprising: transmitting the workflowdataset as a first workflow dataset to the second device, wherein thefirst device is configured to display the animation for a duration oftime before the second device is configured to display the animation;and transmitting the workflow dataset as a second workflow dataset tothe first device at a delayed transmission start time configured tocompensate for the duration of time difference between displayingoperations of the first device and of the second device.
 16. Thenon-transitory computer-readable medium of claim 13, wherein theworkflow dataset is configured to cause the second device to: display,on a first portion of the second display, the first image data depictedon the first display; and display, on a second portion of the seconddisplay, the animation independent of the first image data depicted onthe first display.
 17. The non-transitory computer-readable medium ofclaim 16, the operations comprising: receiving a second input from thesecond device corresponding to a manipulation of the second portion ofthe second display; generating an adjusted workflow dataset based atleast in part on the second input from the second device; andtransmitting the adjusted workflow dataset to the first device and tothe second device, wherein the first device is configured to update thefirst display with a representation of the manipulation in response toreceiving the adjusted workflow dataset, and wherein the second deviceis configured to update the first portion of the second display with therepresentation of the manipulation in response to receiving the adjustedworkflow dataset.
 18. The non-transitory computer-readable medium ofclaim 17, wherein the adjusted workflow dataset is configured to causethe first device and the second device to respectively update the firstdisplay and the second display with audio feedback placement, hapticfeedback placement, one or more lines, text, one or more overlaidimages, or any combination thereof.
 19. The system of claim 1, whereinthe directional haptic feedback is configured to guide an operator tomove the first device to the third position and orientation.
 20. Thesystem of claim 1, the first device comprising a handheld device.